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ABSTRACT 


In the 2013-2014 school year, we implemented the “flipped classroom” as part of an initiative 
to drive active learning, student engagement and enhanced learning in our school. The flipped 
courses consisted of freshman through senior engineering classes in introductory programming, 
statics/mechanics, mechanical design, bio-thermodynamics, facilities layout/material handling, 
and chemical engineering dynamics and modeling. In the flipped classroom, students watch video 
lectures beforehand to obtain the foundational knowledge and then demonstrate skills during 
class. Our study set out to address the following research questions: 1) Does the flipped classroom 
promote student engagement during class, and does it positively impact the classroom environ¬ 
ment?, 2) Is the flipped classroom associated with increased student achievement and learning of 
content?, and 3) What strengths, benefits, and drawbacks do students perceive with the flipped 
classroom? To address these, we used a mixed methods approach, including environment and evalu¬ 
ation surveys, instructor interviews, exam and homework results, video access data, and structured 
classroom observation. Based on our use of the College and University Classroom Environment 
Inventory (CUCEI), we found evidence that flipped instruction can positively impact the classroom 
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environment. We also used a behavioral observation protocol - the Teaching Dimensions Observa¬ 
tion Protocol (TDOP) - to assess student engagement and involvement during class. We compared 
our results to a national TDOP study of 58 lecture-based STEM classrooms, formally demonstrating 
the advantages of our flipped classrooms. Behaviors such as student discussion and questions and 
problem solving were significantly higher in our flipped classrooms (p<0.0001). Our pre-flip versus 
flip exam and homework results were mixed from a statistical improvement standpoint. However, 
based on instructor interviews we noted enhanced higher-order skills such as problem solving and 
deeper engagement and proficiency in some courses and with some students. Unfortunately, we 
encountered challenges with our freshman and seniors. The great majority of freshmen did not use 
the videos for first-time instruction. The seniors expressed resistance to and dissatisfaction with this 
instructional change. Both freshmen and seniors rated their classroom environments statistically 
lower than the sophomores and juniors did. We uncovered other instances in the literature of these 
challenges. Nonetheless, we believe that flipped instruction is a valuable approach for promoting 
engagement and learning. We discuss lessons learned, including the need to educate students about 
the expectations of the flipped classroom. 

Key words: Flipped classroom, inverted classroom, assessment, behavioral observation, engagement, 
engineering education 


INTRODUCTION 


Complex topics, which are often encountered in university STEM courses, benefit from active learn¬ 
ing exercises in which students can more fully construct an understanding of the topic [1], However, 
traditional lecturing still occurs in large part in undergraduate STEM courses, and it is emerging as 
an ineffective way to educate students and help them fully understand the material [2-4], To this 
end, the flipped classroom is an approach that enables students to practice and demonstrate their 
skills during class in the presence of the instructor or teaching assistant (TA) without loss of content 
coverage. With the flip, students’ first exposure to the content is typically via online videos, while 
problem solving and skills application is undertaken during class with mentors present [5-7]. Such an 
active learning approach requires students to be engaged and involved in their learning and reduces 
the amount of passive class attendance [8-9], Top educators have emphasized that meaningful 
learning occurs when students discuss, analyze, solve, apply, and otherwise get involved in their 
coursework [10], Studies have shown that active or interactive learners exhibit significantly higher 
outcomes and gains compared to passive learners in regards to problem solving, time to mastery, 
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and conceptual understanding [9,11], A recent meta-analysis of studies that compared active learn¬ 
ing to the traditional lecture showed that exam performance increased by half a standard deviation 
(on average) in STEM courses that contained active learning. In addition, the average failure rate 
was 34% with traditional lecturing - but only 22% with active learning [4], Flipped courses also lead 
to benefits such as increased student-teacher interaction, student teaming, individualized support, 
self-paced learning with "pause and rewind” capability, increased problem solving and concept 
application, and flexibility for those who cannot attend class for legitimate reasons such as varsity 
sports travel or job interviewing [12-13], 

Given these advantages, Pitt’s Swanson School of Engineering began to formally promote the 
flipped classroom across its various programs in the fall of 2013 with the assistance of its Engineer¬ 
ing Education Research Center (EERC). A school-wide approach has the benefit of promoting this 
pedagogy from the beginning and then throughout the students’ undergraduate careers. The school’s 
objectives with the flipped classroom were the following: 1) enhance in-depth learning and achieve¬ 
ment of higher-order skills in Bloom’s taxonomy, 2) enhance student engagement in learning, and 3) 
better utilize the school’s state-of-the-art instructional facilities and technology to support active 
learning. Interestingly, in a similar manner, approximately 1,100 faculty members from the US and 
Canada indicated that their top motivations for flipping the classroom included increasing student 
engagement (79%) and improving student learning (76%) [14], In another recent survey by the Na¬ 
tional Center for Case Study Teaching in Science, instructors indicated they flip material to increase 
interaction with students, offer flexibility, and involve and engage students in their education [15], 

The courses that we flipped are described in Table 1. Over 1800 students took these courses in 
the four semesters between fall 2013 and fall 2014 inclusive. 

Across these various courses, the motivations and goals of the instructors for flipped instruction 
aligned well with the overall school-level objectives, as shown in Table 2. 

Research Questions and Contributions 

Our overall school-level objectives provided the basis and direction for our research questions. 
Our three research questions were as follows: 

1. Does the flipped classroom promote student engagement during class, and does it positively 
impact the classroom environment? 

2. Is the flipped classroom associated with increased student achievement and learning of con¬ 
tent? 

3. What strengths, benefits, and drawbacks do students perceive with the flipped classroom? 

To address these research questions, we developed a customized, comprehensive assessment plan 

consisting of direct and indirect measures that aligned directly with the objectives of the initiative. 
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Course 

Approx. 
Students 
Per Year 

Level 

Topics 

Introductory Engineering 
Programming 

700 

Freshman/Transfer 

Programming in MATLAB and C for engineering 
problem solving. 

Statics/Mechanics 

160 

Sophomore 

Statics and strength of materials; analysis of internal 
stresses, strains, and displacements. 

Intro to Mechanical 
Engineering Design 

200 

Sophomore 

Fundamentals of mechanical design; concept generation, 
graphical communication, CAD with SolidWorks, and 
material selection. 

Bio-thermodynamics 

90 

Sophomore 

Mass balance, conservation of energy, entropy, 
thermodynamic relations, applications to physiologic 
systems. 

Facility Layout/ Material 
Handling 

70 

Junior 

Space requirements, layout types and algorithms, facility 
location problems, warehousing, and material handling 
methods. 

ChE Dynamics, Modeling 
& Control 

100 

Senior 

Chemical engineering process and systems modeling, 
dynamics, and control. 


Table 7. Courses Flipped. 


These measures included course-embedded assessments such as exams and assignments, interviews 
and discussions with instructors, student perception surveys, structured classroom observation, and 
analysis of video usage data. The perception surveys included a classroom learning environment and 
a flipped classroom evaluation survey. For the evaluation survey, we developed a grounded-theory- 
based, qualitative framework that can be used by other education researchers to code open-ended 


Course 

Instructor Goals 

Introductory Engineering Programming 

Enhance programming skills through increased hands-on application and 
support during class. 

S tatic s/Mechanic s 

Enhance mastery of fundamental skills. 

Engage students beyond rote learning. 

Promote a physical intuition for the problem. 

Enable more in-depth teaming during class. 

Intro to Mechanical Engineering Design 

Introduce more design activities during class. 

Increase teaming and group discussion during class. 

Increase oversight of the student design process. 

B io-thermodynamics 

Enhance understanding and retention of material via increased practice and 
application in the classroom. 

Facility Layout/ Material Handling 

Use class for difficult topics; enhance understanding. 

Use class to engage students in problem solving and provide support. 

ChE Dynamics, Modeling & Control 

Promote “deep thinking.” 

Enhance analysis, problem solving, synthesis, modeling, and computation/ 
simulation. 


Table 2. Instructors’ Goals with Flipping. 
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student responses about the perceived benefits and drawbacks of the flipped classroom. Based 
on a review of the literature, we believe our assessment plan and its implementation makes a novel 
and significant contribution to the literature that others can use to assess their flipped classrooms. 

We believe our article is fairly unique in describing a school-wide implementation of flipped 
classes from multiple engineering disciplines from the first through senior years. Taking a school¬ 
wide approach has the advantage of introducing the method early to students, involving faculty 
from the freshmen through senior levels in the instructional changes experienced by students, 
and being able to gather multiple faculty perspectives simultaneously that can be used to better 
integrate the undergraduate curriculum. The school-wide initiative also enabled the collection of a 
large, freshman-through-senior sample of assessment data under a common evaluation plan. This 
dataset could be used to compare results for different academic levels and provide aggregate results 
based on large samples. Our review of the literature uncovered similar experiences with freshmen 
and seniors at other schools; therefore, our work adds to the depth of the flipped classroom results 
for these special student subsets. 


LITERATURE REVIEW 


Flipped Classrooms and Implementations at Other Institutions 

The flipped classroom represents a shift from an instructor-centered learning environment to 
an environment focused on the student [16], Traditional lecture tends to be instructor-centered, 
with students often missing points made by the instructor due to the pacing or being unable to 
transcribe the words of the instructor fast enough [17-18], The flipped classroom, however, places 
“lecture” under the control of the student via pre-recorded videos or readings that are accessed 
before class; and during class, there is time for students to reflect on problems and apply the con¬ 
cepts being taught [17], 

The flipped classroom represents not only an inversion of typical classroom and homework activi¬ 
ties but also an inversion of expectations on students, including a shift away from passive participation 
[19]. The flipped classroom places more responsibility and expectations of active involvement on the 
student and, based upon the literature, has had mixed acceptance among students, possibly due to 
these new expectations [19], The flipped classroom also entails a change in the faculty member’s role 
in the classroom from that of a content provider to a supporter of students’ analysis and mastery 
of material [17, 19], This involves instructing at the higher levels of Bloom’s taxonomy, uncovering 
students’ misunderstandings, correcting the misconceptions, and answering students’ questions on 
the spot [16,19], The flipped classroom often involves more preparation time for faculty upfront, and 
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given the large change that occurs for both instructors and students, some have proposed flipping 
only a portion of the class sessions within a course [17, 20], 

In reviewing the literature for STEM courses that have been flipped by others, we determined 
that the flipped classroom has been implemented in similar engineering courses at other universi¬ 
ties. These include mechanical engineering courses in statics, numerical methods, introductory 
mechanical design, and electronics instrumentation [21-25], Two universities have used online stat¬ 
ics materials for both flipped and blended classrooms [21-22], These web-based statics materials 
were developed as part of the Open Learning Initiative (OLI) at Carnegie Mellon [26], There have 
also been flipped implementations in an industrial engineering course in work design [27] as well 
as a biomedical engineering course focused on conservation of energy [28], The flipped classroom 
has also been used in both senior-level and first year engineering courses. We found senior-level 
implementations in a mechanical engineering control systems course, an electrical engineering 
signal-processing course, and a software engineering course [29-32], Although we did not un¬ 
cover chemical engineering implementations in the literature, we did find a large collection of over 
1,000 chemical engineering videos that have been successfully used by students to learn chemical 
engineering concepts [33-34], We did uncover successful flipped classroom implementations in 
lower-level chemistry labs and organic chemistry, which are required courses for our chemical en¬ 
gineers [35-36], First-year engineering offerings that have been flipped elsewhere include courses 
in computer applications and problem solving using Excel, MATLAB, and C [37-40], introductory 
engineering [41], and introductory engineering design [42], 

In examining the direct assessment outcomes at other schools, the flipped versus non-flipped 
results have been mixed. At the University of Puerto Rico, an inverted classroom was implemented for 
several sections of a statics course. On the Concept Assessment Tool in Statics (CATS), students in 
the inverted sections scored statistically higher than students in the traditional sections (p= 0.0076) 
[25], At the University of North Dakota, several undergraduate mechanical engineering courses 
were flipped, including introductory design, introductory mechanics, and numerical methods [23], 
In numerical methods, the traditional section had the highest achievement, with 82% earning a C 
or better. This compared to 72% in the flipped section. Over an extended period at Penn State in 
an environmental engineering course, there was no statistical difference in final exam scores across 
six semesters in which the course had been taught both traditionally and in flipped mode [43], 
Interestingly, in the recent faculty survey discussed in the introduction, only one-half (55%) noted 
evidence of improved student learning [14], 

Mixed reports can also be found relative to freshmen. At Ohio State, a significant change was not 
found in freshmen final exam grades [39], At the University of Cincinnati, there was a significant 
improvement on only three of seven problems on the final exam, and there was no change on three 
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problems and a significant drop on one [37], In a comparison of achievement in senior courses, 
students in a flipped control systems course performed statistically better or equivalent to their 
traditional-course counterparts [29-30], In a signal processing course, there was a clear improve¬ 
ment in student performance with the flipped classroom, with an overwhelming majority performing 
at a high level on the final exam, as never seen before [31], 

Results from the literature indicate that some students prefer the flipped classroom while others 
do not. Fifty-four percent of the North Dakota on-campus mechanical engineering students pre¬ 
ferred the flipped format [23], Likewise, in an electronics instrumentation course taken by mechani¬ 
cal engineers, 56% indicated a preference for online videos versus traditional lectures [24], There 
was some dissatisfaction at North Dakota that the instructor had not given “live” lectures in the 
campus-based courses [23], Some instructors have considered the desirability and need for upfront 
mini-lectures in the flipped classroom [30, 44], There are mixed reports in the literature of fresh¬ 
man preferences for the flipped classroom. In a computing course, survey respondents expressed 
a lack of preference for the fully-flipped classroom, with only 13% preferring it. The large majority 
(i.e., 72%) expressed a preference for a partially inverted classroom [38], Yet, freshmen in another 
flipped engineering course in design rated the course as significantly more effective overall than did 
freshmen in the traditional course [42], With flipped first-year engineering courses, a low level of 
classroom preparation is a theme that has been discussed in the literature [41, 45], With seniors or 
upper-division courses, initial student resistance and frustration with inverted instruction has been 
discussed in the literature [29-31], These outcomes and themes from the literature also surfaced in 
our own overall experiences with freshmen through seniors in the flipped classroom. 


METHODS 


Faculty and Course Development 

Preparations for the flipped classroom began approximately six months before the start of 
the semester. In the spring of 2013 the EERC formed a learning community to guide school-wide 
preparations [46], The community was comprised of various engineering faculty who were flipping 
courses, the assessment analyst, and the IT staff involved in the video creation and editing. Learn¬ 
ing community meetings were held approximately every two months, in which various topics were 
discussed, including challenges regarding students and video development, assessment activities, 
classroom logistics, active learning techniques, and the instructors’ goals and motivations. One of 
the challenges often discussed was students’ use of the videos. We eventually came to the conclu¬ 
sion that we needed to better prepare students for our expectations of their responsibilities with 
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the flipped classroom. This culminated with the creation of a “how to” video for students on being 
successful in the flipped classroom, which can be shared with students on day one. This video can 
be accessed at the following link: “How To” Flipping Video. 

To further prepare the faculty, the EERC organized a one-day seminar in the spring of 2013, led 
by engineering faculty from another institution having experience with the flipped classroom. Also, 
one of the learning community members piloted three “flipped” lectures during the spring 2013 
semester, prior to full implementation the following year. The information he was able to gather on 
the video recording, day-to-day assessment needs, and course and classroom logistics was very 
helpful to both him and others in the community. 

Each instructor began creating his video lectures in the semester prior to the implementation 
semester. The instructors were provided with small school grants to support the development of 
their flipped courses. The lectures were recorded in small modules using the Camtasia software with 
the assistance of the IT staff [47], Across the six courses, the videos ranged in average length from 
seven to eleven minutes. We aimed to keep the videos short (i.e., under 15 minutes) and modular, as 
recommended [7,12], As discussed in the literature, ten minutes may be an upper limit on students’ 
attention spans [48], To give an example of the modularity of the lectures, Table 3 contains sample 
video titles from the freshman programming course. 

Assessment 

Teaching Dimensions Observation Protocol (TDOP) 

To assess engagement, involvement, active learning, and classroom dynamics and usage, we 
observed two class sessions in each flipped semester between fall 2013 and fall 2014 using the 
Teaching Dimensions Observation Protocol (TDOP) [49], We were able to collect pre-flip observa¬ 
tion data for two courses in the spring of 2013. We observed class sessions at approximately the 
one-third and two-thirds points in the term. In using the TDOP, the total class period was divided 
into a series of consecutive five-minute observation segments, or windows. Thus, in a given class 
period, multiple five-minute time segments existed and were observed. For example, a 50-minute 
class would have had 10 observation segments. In each segment, various activities and practices 


Branching - “If' Basics Looping — For Loops 
Branching — “If’ Example Looping — Avoiding Infinite Loops 
Looping - While Loops Application - Random Numbers 

Table 3. Modularity of Videos (Freshman Programming Course). 
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within the protocol were recorded when observed. The dimensions that describe these activities 
and practices are as follows: 1) teaching methods, or how information is disseminated and learning 
is accomplished during class; 2) pedagogical moves, pertaining in part to teaching style and strat¬ 
egy; 3) questioning between instructors and students; 4) cognitive engagement by students, such 
as problem solving; and 5) instructional technology usage. The developers of the TDOP have used 
both a five-minute and a two-minute window and initially reported inter-rater reliability using the 
five-minute window [49], Based on personal communication with the developer of the TDOP, the 
two-minute window provides more granular data, as more happens during a five-minute period of 
time versus a two-minute period. However, the two-minute window places more demands on the 
observer and may decrease his/her ability to record notes and casually and comprehensively assess 
classroom happenings and the environment [50], 

When we were able to collect pre-flip observation data, we assessed changes (pre-flip to flip) 
in the occurrence of certain behaviors of interest, such as student dialogue or problem solving, 
using Fisher’s Exact test. For those courses in which we were not able to collect pre-flip data, we 
compared our results to those of a recent TDOP study of 58 STEM classrooms to measure the rela¬ 
tive amount of active and interactive learning in our flipped classrooms. Either one or two trained 
observers performed the classroom observation using the TDOP. When two observers performed 
the observation, they discussed any differences in assigned codes afterwards until a consensus was 
reached. Our overall inter-rater reliability statistic for use of the TDOP was Cohen’s k=0.86, with the 
five dimensions ranging from 0.70 to 0.92. Values of k above 0.75 suggest strong agreement beyond 
chance; values between 0.40 and 0.75 indicate fair levels of agreement above chance [51], These 
statistics are based on a total of 80 five-minute observation segments across four different courses. 

College and University Classroom Environment Inventory (CUCEI) 

We distributed the classroom environment survey, formally known as the College and University 
Classroom Environment Inventory (CUCEI), to the students at approximately the two-thirds mark 
of the semester. This instrument reliably evaluates seven psychosocial dimensions of the classroom 
and has been used previously in flipped classroom research [6, 52], Several of the dimensions are 
particularly relevant to the flipped classroom, including involvement, student cohesiveness, indi¬ 
vidualization, personalization, and innovation. There are seven questions per dimension, and each 
question has a scale of 1 to 5, with 5 being most desirable. For those courses in which we were able 
to collect pre-flip environment data, we assessed changes in the classroom environment using f-tests 
and Cohen’s d effect size calculations. The effect size represents the extent of the difference between 
two groups. Cohen defined effects as small (d= 0.20), medium (d= 0.50), or large (d= 0.80) [53]. 
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Flipped Classroom Evaluation Survey 

We also administered the flipped classroom evaluation survey at the two-thirds point of the term 
to assess student attitudes towards and perceptions of the flipped classroom, including benefits 
and drawbacks perceived. Our flipped classroom evaluation survey was modeled upon the work of 
Leicht, Zappe and colleagues, who used student perception instruments in a flipped course with 
Penn State undergraduates having sixth-semester standing in architectural engineering [5, 7], We 
extended this survey via input from our own faculty. A trained coder, who was a junior engineering 
student, conducted a content analysis of the open-ended questions on benefits and drawbacks 
perceived. A second coder, who was the assessment analyst for the project, served to establish inter¬ 
rater reliability by coding a sample (i.e., approximately 30%) of the responses [54], The frameworks 
used for the coding were developed by the assessment analyst based on a grounded, emergent 
qualitative analysis of all student responses [54], The inter-rater reliability scores achieved based 
on Cohen’s Kappa were k = 0.75 for the benefits analysis and k = 0.83 for the drawbacks analysis. 

Assessment of Learning and Achievement 

To assess achievement in the flipped classroom, we compared direct assessment data, including 
homework and exam scores, between the pre-flipped and flipped versions of the course using an 
analysis of covariance with the pre-course cumulative GPA or SAT score as the covariate, or control 
variable. Pre-flipped versions of the course had been taught in previous semesters. The instructors 
were also interviewed after the course to discuss gains and outcomes that may not have been ap¬ 
parent in the direct assessment results. Finally, to understand the relationship between prepara¬ 
tion and achievement in the flipped classroom, we performed a correlational analysis between the 
number of videos accessed and the final course grade. The number of videos accessed was based 
on web analytics data. 


RESULTS 


As discussed, we performed a variety of assessment activities to evaluate the effectiveness of 
our flipped classroom initiative and to address our research questions. We used the structured 
classroom observation and the classroom environment survey to address question 1 regarding 
student engagement during class and the classroom environment. Exam and homework results 
(pre-flip versus flip), instructor interviews, and a correlational analysis of video usage and course 
grades were used to investigate question 2 about the impact of the flipped classroom on achieve¬ 
ment and learning of course content. With our flipped classroom evaluation survey, we identified 
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students’ perceptions of the benefits and drawbacks of the flipped style to investigate question 3. 
Based on a review of the literature, we also provide a comparison of our implementation to other 
flipped implementations nationally. 

RQi: Student Engagement During Class: Observation 

Structured observation of our flipped classrooms using the TDOP was conducted to evaluate 
student engagement and involvement during class. We were also able to observe two of the courses 
before they were flipped. A quantitative and qualitative description of each classroom based upon 
this observation is provided in this section. Table 4 displays the percentage of five-minute obser¬ 
vation segments in which each classroom element of interest was observed for each course. In 
addition, we compared four of our flipped courses against a national 2012 TDOP study that used 
two-minute observation windows for lecture sessions of STEM classes. The courses in Table 4 are 
those for which we did not have pre-flip observation data. 

Flipped Classroom Descriptions and Comparison to National TDOP Study 

Bio-thermodynamics: Our bio-thermodynamics course was characterized by a large amount of 
instructor-to-student interaction as well as interaction among the students. The instructor circulated 
during 82% of the five-minute observation segments (MOV) as he frequently posed questions. His 
style was very interactive; he persisted until he received answers to his questions. The students 
worked in small groups (SGW) and discussed course content (ART) in 66% of the segments. These 
discussions centered on problem solving efforts (PS), divergent responses to challenging clicker 
questions (CL), or team quizzes (A). 

Chemical Engineering Dynamics, Modeling and Control: In the chemical engineering course, 
there was a large amount of student activity and interactivity. In 65% of the segments, students 
actively discussed course material (ART). The students solved problems in 44% of the segments 
(PS), often working individually and/or with software at the desk (DW) while assisting one another. 
At other times, the students specifically worked in small groups (SGW) on their class projects or 
to resolve disagreements on a clicker question. This was complemented by the instructor and TAs 
circulating (MOV) to provide support as the students asked questions or sought help (SCO) in 37% 
of the segments. Assessment (A) was accomplished using clickers (CL). 

Freshman Programming: The freshmen programming class was also characterized by a large 
amount of active student work, although it was very active even before flipping the course. In 56% 
of the segments, students actively worked at the desk (DW) with MATLAB or C. This was often ac¬ 
companied by instructor demonstration of coding tasks from the podium computer. Active problem 
solving (PS) with discussion among the students (ART) occurred in 40% and 32% of the segments, 
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respectively. This was complemented by the instructor and TA circulating among the students (MOV), 
typically to provide help, in 45% of the segments. Student questioning to seek assistance (SCQ) 
occurred in 47% of the segments, which was likely encouraged by the circulation of the instructor 
and TA. Students actively worked on assessment quizzes in 32% of the segments; the quizzes in¬ 
volved analyzing or writing code during class. The students were specifically asked to work in small 
groups in 29% of the segments. During the fall 2014 semester, the students were asked to practice 
coding (in small groups) during class more than ever before, which was enabled partly by switching 
to coding-based quizzes. Thus, our freshman programming class has become more hands-on and 
aimed at providing needed programming experience for students. 

Mechanical Engineering Design: The mechanical engineering design course represented an out¬ 
standing implementation of the flipped classroom. In short, the teaching assistants were in nearly 
constant demand during class and provided as-needed support to the students’ use of SolidWorks. 
Specifically, in 97% of the segments, the TAs circulated to provide support (MOV) and answer the 
students’ questions (SCQ). During nearly the entire class period, students actively worked with 
SolidWorks at their desks (DW) to solve the design problem (PS) posed to them. These problem 
assignments were generally due at the end of the two-hour period. At the same time, the students 
discussed the work and assisted one another (ART). 

TDOP Comparison Study: The national TDOP study was based on lecture-based classrooms 
and as such does not provide a benchmark per se for our active, flipped classrooms. However, the 
national study does provide some perspective and a point of reference for our observation results. 
As can be seen in Table 4, our flipped classrooms were active and interactive compared to those 
in the TDOP study. The national study occurred with of 58 math and science faculty in three public 
research universities and involved 38% upper and 62% lower division offerings [55], 

Each element in our flipped classroom that was very significantly higher compared to the national 
study with p<0.0001 is marked with an asterisk (*). These elements would also remain significant 
after correcting for multiple comparisons using Bonferroni’s adjustment [56], This demonstrates 
that flipped instruction drives student engagement and involvement in learning and was successful 
in promoting better utilization of our instructional facilities designed for team-based active learn¬ 
ing. Similar to the national TDOP study, Finelli and Daly also noted a small number of classroom 
observation segments (i.e., 9%) that involved active learning when assessed across 26 engineering 
courses using a variation of the TDOP [57]. 

In a personal conversation with one of the developers of the TDOP, he pointed out that it’s pos¬ 
sible a five-minute observation window could result in higher percentages compared to a two-minute 
window in certain cases [50], For example, if problem solving (PS) was observed just once in ten 
minutes using the two-minute observation window, its frequency of occurrence would be 20%. 
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% of Observation Segments 



Practices Observed 

STEM 

Comparison 

Study 

Bio 

Thermo 

Dynamics 

Chemical 


Mechanical 

Engineering 

Design 

Classroom 

Element 

Description 

Engineering 

Dynamics/ 

Modeling 

Freshman 

Programming 

MOV 

Instructor circulates in 
classroom 

7% 

82%* 

37%* 

45%* 

97% * 

SGW 

Small group work/discussion 

9% 

66%* 

21% 

29%* 


ART 

Student articulation/discussion 

9% 

66%* 

65%* 

32%* 

86%* 

PS 

Problem solving 

12% 

34%* 

44% * 

39%* 

97 %* 

CL 

Use of clickers 

5% 

30% * 

14% 



A 

Assessments (quizzes) 

5% 

28%* 

14% 

32%* 


SCQ 

Students ask question or request 
assistance 

9% 

23% 

37%* 

47%* 

97 %* 

DW 

Students actively work at desk/PC 

6% 


44% * 

56%* 

98%* 


*p < 0.0001 

Table 4. Comparison to 2012 National STEM TDOP Study (Two-Minute Window). 


However, using the five-minute window, the frequency of occurrence would be 50%. Granted, in 
this case, problem solving (PS) occurred rather sparsely during the ten minutes. If PS had occurred 
continuously during the ten minutes, then the frequencies of occurrence would have been the same 
at 100%. In the flipped classroom, our students often worked for a substantial period (i.e., longer 
than two minutes at a time) on problem solving in groups or individually with student discussions 
taking place. Nonetheless, we obtained earlier comparison data from the developer of the TDOP, in 
which a five-minute observation window had been used. This data was collected during the spring 
of 2010 and involved 57 math and science instructors at three large research universities [49], We 
used this data as a second comparison, as shown in Table 5. However, the developer cautioned that 
the TDOP was at an early stage of development when he used the five-minute window and was a 
much different protocol compared to the present instrument. Since several of the codes in Table 4 
did not exist in the earlier protocol, we could only compare certain elements, such as SGW, PS, CL, 
and DW. Thus, while comparing our five-minute-window results against the two-minute-window 
study was not optimal as described, a comparison using the earlier five-minute-window study was 
likewise difficult, given the differences with the current instrument. 

In comparing tables 4 and 5, the most notable difference was related to the problem solving (PS) 
element. Using the five-minute comparison window as shown in Table 5, the comparisons of the 
Bio-thermodynamics and Chemical Engineering courses did not represent significant differences, 
although they did with the two-minute window comparison in Table 4. However, based upon either 
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% of Observation Segments 



Practices Observed 

STEM 

Comparison 

Study 

Bio 

Thermo 

Dynamics 

Chemical 

Engineering 

Dynamics/ 

Modeling 

Freshman 

Programming 

Mechanical 

Engineering 

Design 

Classroom 

Element 

Description 

SGW 

Small group work/discussion 

5% 

66%* 

21% 

29%* 


PS 

Problem solving 

31% 

34% 

44% 

39% 

97%* 

CL 

Use of clickers 

6% 

30%* 

14% 



DW 

Students actively work at desk/PC 

5% 


44%* 

56%* 

98%* 


*p < 0.0001 


Table 5. Comparison to 2010 TDOP Study (Five-Minute Window). 


the two-minute or five-minute comparison study, our flipped classrooms exhibited a large degree 
of active learning and student interaction, illustrating what is possible with a flipped classroom. 

Pre-Flipped vs. Flipped Classroom Observation 

Statics/Mechanics: Statics/Mechanics was one of the two courses we were able to observe be¬ 
fore it was flipped. The pre-flip statics course did not contain elements of active learning, such as 
small group work (SGW), student discussion (ART), or problem solving (PS), as shown in Table 6. 
However, in the flipped course observed multiple times between the fall of 2013 and the fall of 2014, 
a change occurred in active learning, as shown in the table. The values for the flipped classroom are 
an aggregation of the data collected over all four semesters. The level of interactivity also increased, 
with a greater percentage of segments containing student questions or requests (SCQ), instructor 



Practices Observed 

% of Observation 
Segments 

Difference 

P 

Classroom 

Element 

Description 

Pre 

Flip 

Flip 

ART 

Student articulation/discussion 

0% 

42% 

0.007 

SGW 

Small group work/discussion 

0% 

42% 

0.007 

MOV 

Instructor circulates in classroom 

0% 

41% 

0.013 

SCQ 

Students ask question or request assistance 

0% 

36% 

0.029 

PS 

Problem solving 

0% 

35% 

0.029 

A 

Assessments (quizzes) 

0% 

11% 

0.594 


Table 6. Pre Flip vs. Flip Observation for Statics/Mechanics. 
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circulation (MOV), and student discussion (ART). Team quizzes were also given (A); they were a 
form of active learning and a means to drive accountability with the videos. 

This course actually evolved over time in terms of the ideals of the flipped classroom. During the 
first flipped semester, the number of segments containing small group work (SGW) and student 
discussion (ART) as the instructor circulated (MOV) was low at 10%. During the second semester, 
these elements each occurred in 47% of the segments, while problem solving (PS) occurred in 29%. 
By the third semester, all four of these key elements each occurred in 64-67% of the observation 
segments. During the fourth semester, the percentages were not as high as during the second or 
third semesters but higher than in the first semester. 

We compared the pre-flip to the flip percentages in Table 6 using Fisher’s Exact test; Fisher’s test 
can be used in lieu of a z-test of proportions when the numerators are small. The increases in ART 
and SGW were the most significant (p<0.01) and would remain significant at a=0.05 if corrected 
for multiple comparisons using Bonferroni’s adjustment. 

Facility Layout: In the pre-flipped facility layout course, problem solving (PS) with student dialogue 
(ART) each occurred in only one of the observation segments (7%), indicating the opportunity for 
increased active learning. This opportunity was met by flipped instruction, with problem solving and 
student dialogue each occurring in 56% of the segments. This was complemented by the instructor 
and TA circulating to provide assistance (MOV), which occurred in 63% of the segments. Student 
questions and requests (SCQ) increased from 13% to 56%, which was likely encouraged by the cir¬ 
culation. Based on Fisher’s Exact test, the most significant increases were in instructor circulation 
among the students, problem solving, student discussion as shown in Table 7; these would remain 
significant at a=0.05 if corrected for multiple comparisons. 

The increases within our statics/mechanics and facility layout courses (pre-flip to flip) in student 
discussion (ART), small group work (SGW), problem solving (PS), and instructor circulation (MOV) 



Practices Observed 

% of 

Observation 

Segments 

Difference 

P 

Classroom 

Element 

Description 

Pre 

Flip 

Flip 

SCQ 

Students ask question or request assistance 

13% 

56% 

0.023 

PS 

Problem solving 

7% 

56% 

0.006 

ART 

Student articulation/discussion 

7% 

56% 

0.006 

MOV 

Instructor circulates in classroom 

0% 

63% 

0.0002 


Table 7. Pre Flip vs. Flip Observation for Facility Layout. 
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to answer questions and provide help illustrate the impact that flipped instruction can have on the 
engagement, interactivity, and student involvement in engineering classrooms. 

RQ1: Flipped Classroom Environment: Environment Survey 

It is very important to understand students’ perceptions of their experiences in the classroom 
and the psychosocial environment in which they learn in addition to assessing their performance 
[62], To this end, we assessed the psychosocial dimensions of our flipped classroom as defined in 
Table 8 using the College and University Classroom Environment Inventory (CUCEI). We received 
a total of 793 responses from students in our six flipped courses between fall 2013 and fall 2014 
(inclusive), representing a 43% response rate. Across these courses, the personalization dimen¬ 
sion scored the highest of the seven dimensions, with a dimension mean of 3.88 on the five-point 
scale, with a score of 5 being most desirable. This dimension relates to the interaction between the 
students and their instructor and is thus a perceived strength of the flipped classroom. This is a 
good outcome for the flipped classroom, as Astin’s large-scale correlational study of over 20,000 
students found that frequent interaction between faculty and students was critically important to 
student development and satisfaction [58-59], 


Dimension 

Definition 

Pitt 

Engineering 

Flipped 

Courses 

Statistics 

Flipped 

Course 

Chemistry 

Tutorial 

Course 



M 

M 

M 

Student Cohesiveness 

Students know & help one another 

3.04 

3.00 

2.87 

Individualization 

Students can make decisions; treated 

2.64 

2.58 

2.59 


individually or differentially 




Innovation 

New or unusual class activities or 

2.99 

3.08 

2.84 


techniques 




Involvement 

Students participate actively in class 

3.29 

- 

3.03 

Personalization 

Student interaction w/ instructor 

3.88 

4.13 

3.26 

Satisfaction 

Enjoyment of classes 

3.39 

- 

3.40 

Task Orientation 

Organization of class activities 

3.74 

3.51 

3.31 


n 

793 

23 

257 


Pitt SD values: Cohesiveness 0.804; Individualization 0.475; Innovation 0.565; Involvement 0.633; Personalization 
0.752; Satisfaction 0.915;Task Orientation 0.661 

Statistics SD values: Cohesiveness 0.71; Individualization 0.56; Innovation 0.60; Personalization 0.75; Task 
Orientation 0.69 

Chemistry SD values: Cohesiveness 0.90; Individualization 0.62; Innovation 1.02; Involvement 0.57; 
Personalization 0.70; Satisfaction 0.78; Task Orientation 0.70 

Table 8. CUCEI Comparisons. 
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In a post-course interview, our bio-thermodynamics instructor shared that in his flipped classroom, 
students were less hesitant to interact with him and ask questions during class. The individualiza¬ 
tion and innovation dimensions scored lowest and below the average value of 3.00, with dimension 
means of 2.64 and 2.99, respectively. Thus, our respondents did not perceive notable individual or 
differential treatment or innovative techniques, which are two key goals of flipped instruction; thus, 
these dimensions may represent opportunity areas. An internal consistency reliability analysis of 
our flipped classroom environment data showed Cronbach’s alpha reliabilities above or near 0.70 
for five of the seven dimensions [60], 

Comparison to Other Classroom Environments 

We had the goal of comparing our flipped classroom environment data against a large-scale 
study, as we had done with the TDOP data. However, based on a search of the literature and per¬ 
sonal communication with the developer of the CUCEI, we were unable to identify such a study 
[61], The developer of the CUCEI had recently completed a literature review on classroom learning 
environments discussing over 40 years of research with STEM classrooms [62], However, we did 
find two smaller-scale CUCEI studies involving courses with similar classroom formats. Compared 
to a flipped statistics course at another US university, our courses compared closely, as shown 
in Table 8. Although the CUCEI instrument used at this other university differed somewhat from 
Fraser’s original instrument, we determined the questions to be sufficiently similar and suitable 
for comparison [6, 63], Also, two of the dimensions in Fraser’s instrument - satisfaction and in¬ 
volvement - were not measured in the statistics study. Although three of our dimension means 
were higher, all five means were statistically equivalent between the two schools based on a 
f-test. This was likely influenced by the small sample size at the other school. A non-parametric 
test would have been preferable; however, we had only summarized (i.e., mean) data from the 
statistics study. From a takeaway perspective, Personalization was also rated highest in the sta¬ 
tistics flipped classroom, while Individualization was also the lowest rated dimension. This may 
be a general result for flipped classrooms. A second, larger CUCEI study was performed on a 
chemistry tutorial class at the University of the South Pacific [64]. This tutorial class was similar 
to our flipped classroom in that it was an interactive class in which students had the chance to 
clarify lecture content and develop scientific problem solving skills under guidance. Our flipped 
classroom environment was significantly better on five dimensions - involvement, personalization, 
and task orientation (p<0.0001) as well as student cohesion and innovation (p<0.005). Thus, we 
compared favorably to this chemistry course having a similar class format. Standard deviations 
are provided so that comparisons can be run by others who may wish to use this instrument in 
a flipped classroom. 
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Comparison of Pre-Flipped and Flipped Environments 

We were able to collect pre-flip environment data for two courses, given the timing of the initia¬ 
tive, and thereby compare these courses in a pre-flip to flipped manner. 

Statics/Mechanics: In the statics course, four of the seven CUCEI dimensions showed a significant 
increase (p<0.0005) pre-flip to flip based on f-tests. These four significantly improved dimensions 
are of key interest to our program evaluation because they are general goals of a flipped classroom 
- individualization, personalization, involvement, and student cohesiveness. Among the significantly 
improved dimensions, individualization was associated with the largest effect size (d= 0.96), although 
students rated it lowest among the seven dimensions in the flipped classroom. Thus, improvement 
was actually greatest for the individualization dimension. Personalization, involvement, and cohe¬ 
siveness had effect sizes of d- 0.95, 0.85, and 0.83, respectively. 

Facility Layout: In the flipped version of the facility layout course, there was a significant increase in 
three of the dimensions based on t-tests - individualization (p<0.0005), personalization (p<0.0005), 
and involvement (p=0.002). Upon adjusting for multiple comparisons using Bonferroni’s correc¬ 
tion, we considered innovation not to be significant (p=0.021). Cohen’s d was found to be large for 
individualization (c/=2.01) and personalization (c/=1.06) and medium for involvement (d= 0.67). In his 
post-course interview, the facility layout instructor noted there was definitely increased opportunity 
to interact with students during a flipped class (as anticipated), and this coincides with the signifi¬ 
cant increase in the personalization dimension. However, in the instructor’s viewpoint, even more 
interaction would have been possible if additional teaching assistants had been available during class. 

RQ2: Achievement and Learning: Direct Assessments and Instructor Interviews 

We directly assessed the impact of the flipped classroom on student learning and achievement 
using examples of student work that offered the best comparison by virtue of being the same or 
nearly the same in the pre-flipped versus flipped versions of the course. Depending on the course, 
these included exams or homework assignments. The instructors for each course remained the 
same over the various semesters considered. Below, we provide direct assessment results from four 
courses, which are summarized in Table 9. 

Introductory Engineering Programming 

For our freshman programming course, we compared results of the midterm and finals exams in 
the pre-flip vs. flipped courses in the sections in which the videos were emphasized by the instructor. 
In the honors variant of the course, which is taught in the fall semester to first-time freshmen, we 
found an increase in the average midterm score (pre-flip to flip) from 86% to 87%. This was based 
on two semesters of pre-flip (n=109) and two semesters of flip (n=150) data. With the final exam, 
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Course 

Direct Assessment 

Significant 

Difference 

(a=0.05) 

Introductory Engineering Programming 

Exam 

(Midterm & Final) 

No 

Intro to Mechanical Engineering Design 

Homework 

(SolidWorks Assignments) 

No 

Bio-thermodynamics 

Exam 

(4 Topic Areas) 

Yes 

Facility Layout/ Material Handling 

Homework 

(Facilities Location Problem) 

Yes 


Table 9. Direct Assessments Summary. 


there was also an increase in the average from 86% to 87%. This was based on two semesters of 
pre-flip (n=109) and two semesters of flip data (n=150). As a baseline comparison of the pre-flip 
vs. flipped honors groups, we examined the Math SAT score and the first-term cumulative GPA. 
This was an end-of-term GPA but was the first (U. of Pittsburgh) GPA available to us. The Math SAT 
score served as a pre-course variable. We found the average first-term GPA to be higher for the flip 
group, although not significantly so (p=0.14), and there was no significant difference in the Math 
SAT scores between the groups (p=0.82). We ran an analysis of covariance (ANCOVA) with each of 
these variables as covariates, or control variables. For the midterm exam with GPA as the covariate, 
the difference (pre-flip to flip) was not significant (p=0.61). The same was true using Math SAT as 
the covariate (p=0.13). For the final exam, with the first-term GPA as the control variable, the dif¬ 
ference (pre-flip to flip) was also not significant (p=1.00). In addition, with Math SAT as the control, 
the difference in final exam scores was not significant either (p=0.29). 

For the non-honors variant of the freshman programming course, which is taught in the spring 
semester of the freshman year, we analyzed two semesters of non-flip (n=421) and one semester of flip 
data (n= 232), based on the data that was available to us. The pre-course cumulative GPA was used as 
a covariate in a pre-flip to flip comparison of the midterm and final exam scores. There was an increase 
in the midterm score (pre-flip to flip) from 83% to 84%, which showed some evidence of a significant 
change (p=0.056) based on the ANCOVA results. With the final exam, there was a decrease (pre-flip to 
flip) from 80% to 79%, which was not significant (p=0.56) based on the ANCOVA results. The pre-course 
cumulative GPA was slightly lower (although not significantly so) for the flipped-classroom students. 

Mechanical Engineering Design 

In the introductory mechanical design course, we compared students’ achievement on their 
SolidWorks take-home assignments using one semester of pre-flip and two semesters of flip 
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performance data. Using pre-course cumulative GPA as a covariate or control variable, we did not 
find a significant difference pre-flip to flip (p=0.41). The pre-flip percentage was 94% (n=177); the 
flipped percentage was 93% (/i=394). The SolidWorks take-home assignments were generally the 
same between the semesters. The average cumulative GPA for the pre-flip group was just slightly 
higher, although not significantly so. 

Despite the statistical equivalence of the two sets of SolidWorks performances, the instructor 
reflected that the students in the flipped section were more “sophisticated” and proficient CAD users. 
He attributed this to the practice time available with SolidWorks in the flipped classroom - students 
had to solve twice as many problems versus in the non-flipped course. In addition, the instructor also 
noted in both a post-course interview and a focus group that students likewise felt more proficient 
and confident with SolidWorks versus in previous years. Others have reported increased student 
confidence and self-efficacy as well [65]. For the first time in teaching this course, his students 
identified SolidWorks as the best part of the course. 

Bio-thermodynamics 

In our bioengineering course, a comparison of exam results in the flipped semester versus in the 
prior four (non-flipped) semesters uncovered significant improvements. The pre-course cumulative 
GPA was used as a covariate in the comparison. In an exam covering four topic areas, students in 
the flipped class performed better on all four topics - concept mastery (p=0.004), conservation of 
mass (p<0.0005), conservation of energy (p<0.0005), and gas expansion (p<0.0005). The latter two 
topics were associated with large effect sizes, with Cohen’s d values of 0.87 and 0.98, respectively. 
The first three topics had pre-flip vs. flipped sample sizes of 224 and 87, respectively; the fourth 
topic had sample sizes of 167 and 87, respectively. As a comparison of the two groups (pre-flip vs. 
flipped), the average pre-course cumulative GPAs were not significantly different, with the average 
for the pre-flip group just slightly higher (p=0.33). 

Prior to flipping, the instructor did not have time during class to be as interactive with the students or 
as persistent in receiving answers to his questions. He felt this was a large contributor to the statistically 
higher scores on the exam, as expressed during an interview. In addition to this outcome, the instructor 
expressed in an interview that the students in the flipped course were much better problem solvers 
than in prior years. In the flipped course, the students had the opportunity to solve more problems; in 
fact, the students were assigned higher-level problems in the flipped course and solved them or made 
great strides in doing so, based on the instructor’s reflection. Based on these results, he plans to pro¬ 
vide better, more challenging problems to the students going forward. In the instructor’s assessment, 
the flipped classroom was overall a good experience for him and the students; in fact, he believes he 
benefitted because it gave him insight into the types of problems he should be providing to students. 
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Facility Layout/Material Handling 

We compared the results of a facilities location homework assignment in the pre-flip versus 
flipped courses. This homework problem remained the same across the semesters. The pre-flip 
average was based on three semesters of data (n=155), and the flipped average was based on two 
semesters (n=148). The instructor pilot-flipped a few course topics in the semester prior to full 
implementation; thus, we had two semesters of flipped data. Using pre-course cumulative GPA as 
a control variable, we found a significant improvement from the pre-flipped to the flipped course, 
with average scores of 81% versus 89%, respectively (p<0.0005). The effect size was medium at 
d= 0.44. As a comparison of the two groups, the pre-course cumulative GPAs were not significantly 
different between the pre-flipped and flipped groups, with the average GPA for the "flipped” group 
just slightly higher (p=0.37). 

Analysis of Video Usage Data 

We analyzed web analytics data from fall 2013 through fall 2014 to determine the number of 
videos accessed by each student. This data was a direct assessment of their participation in and 
preparation with flipped instruction. We were also interested in the relationship between their 
preparation and achievement in the flipped classroom. Unfortunately, the web analytics data has 
a limitation in that it indicates that a video was accessed or launched and not necessarily that 
it was watched in whole or even in part. In addition, students might have watched the videos in 
groups in the dormitories; therefore, not all students might have officially logged in to watch 
a particular video. The sophomore through senior students appeared to have demonstrated 
some responsibility for the self-directed portion of the flipped classroom; however, it appears 
the freshmen did so to a much lesser extent. Based on our experiences, freshmen may require 
additional monitoring with the flipped classroom. Unfortunately, for the bio-thermodynamics 
course, the web-based data was collected in a different manner and could not reliably be 
combined with data from the other courses; therefore, the bioengineering course data is not 
reflected in Table 10. 



Average % 

n (students) 

Freshmen 

15% 

381 

Sophomores through Seniors 

60% 

784 


Table 70. Web Analytics Percentage of Videos Accessed. 
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Unique Videos Accessed vs. Course Grade 



Figure 7. Videos Accessed vs. Final Course Grade (Freshmen). 


For each of the sophomore, junior, and senior-level courses (excluding the bioengineering 
course), we found correlations between 0.36 and 0.43 for the number of unique videos accessed 
and the final course grade (i.e., grade points associated with the final grade). These were sig¬ 
nificantly different from zero (p<0.001). This suggests some relationship between preparation 
and achievement in the flipped classroom. However, even for freshman in those sections in 
which video watching was emphasized by the instructor, this correlation was only 0.01 and not 
significantly different from zero. For this subset of freshmen, the scatter plot of the unique vid¬ 
eos accessed (out of a possible 79) versus the grade points associated with the course grade is 
shown in Figure 1. This plot shows that a small number of freshmen did in fact “fit” the expected 
relationship between videos accessed and the final course grade. However, many students 
who performed well accessed only a portion of the available videos, as indicated by the large 
number of points near the upper left portion of the plot. This raises several questions for future 
research. For those students who were not accessing the videos but still performing well, how 
did they acquire their knowledge? What are the primary methods, behaviors, or characteristics 
of freshmen who do not fit the expected relationship between video usage and achievement in 
the flipped classroom? 

Thus, our preliminary findings suggest continued research in this area, including investigation of 
the factors that contribute to the low correlation between video usage and course performance for 
freshmen, such as differences in student learning styles, base knowledge, other support or social 
structures, and instructor characteristics. An additional research question of interest is the follow¬ 
ing: when in the students’ undergraduate careers do they begin to take greater responsibility for 
watching the videos as part of the self-directed portion of the flipped classroom? 
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In addition, we also analyzed the total number of times the videos were accessed versus the final 
grade and found the correlations to vary widely by course. This analysis accounted for the cases 
in which a video was accessed multiple times by students, for example for reinforcement or study 
purposes. The correlation for the freshman “emphasis sections” in which video watching was em¬ 
phasized was again low at 0.02 and not significantly different from zero. The correlations for each 
of the sophomore and junior courses (excluding bio-thermodynamics) ranged from 0.19 to 0.33. 
Surprisingly, the correlation for the seniors was -0.01 and not significantly different from zero. We 
defined a “distinct” access of a video as one that occurred at least ten minutes after the last access 
of the video by the student. For example, if a student accessed a particular video at both 2:05 PM 
and 2:07 PM on a given day, these would not be counted as two distinct accesses. We considered 
ten minutes to be reasonable given the average length of the videos. 

Although we collected web analytics data on video usage, we also asked the respondents to 
self-report the percentage of videos they watched, given the limitations in the web analytics data 
(previously discussed). Compared to the web analytics data, the self-reported data indicate higher 
video access percentages. However, even in those freshmen course sections in which video watching 
was emphasized by the instructor, the average percentage reported was markedly lower than the 
percentages reported by the sophomores through seniors, as shown in Table 11. Based on a f-test, 
the freshmen watched a significantly lower percentage of videos in comparison (p<0.0005). The 
percentage reported by Penn State upper-level engineering students provides a second point of 
comparison. Despite some differences in emphasis across the freshman sections, all freshmen were 
directed to watch the videos via the course syllabus and website. This indicates freshmen students’ 
lack of responsibility in preparing for their flipped classes. However, some lecturing still occurred in 
our freshman flipped classes with coding demonstrations by the instructor. This may have impacted 
the level of video watching. Ninety-four percent (94%) of responding freshmen from the sections 
in which the videos were emphasized indicated they watched the videos after the class period for 
which they were assigned; only 11% used the videos for first-time instruction as intended. In addition, 
starting in the spring 2014, we asked freshmen about previous programming experience. Interestingly, 



Average % 

n (students) 

Freshmen 

45% 

95 

Sophomores through Seniors 

77% 

321 

Zappe et al. (Penn State) 

92% 

77 


Table 11. Self-Reported Percentage of Videos Watched. 
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only 29% of respondents from the "emphasis sections” were familiar with programming prior to the 
course. Therefore, video watching would seem to have been warranted by these students; however, 
this did not occur prior to class for the most part. 

RQ3 : Perceptions of Benefits and Drawbacks: Flipped Classroom Evaluation Survey 

The students were asked to evaluate the flipped course via a survey designed to provide both 
formative and summative feedback. Approximately 29% provided feedback between the fall of 2013 
and the fall of 2014. One of the questions we asked was “Do you prefer a flipped classroom over a 
traditional lecture class?” The distribution of student responses showed an approximate three-way 
split, with 27% responding yes, 36% responding no, and 36% being unsure. Our seniors, with whom 
we experienced some resistance, responded definitively “no” in 56% of the cases. We compared 
the proportion of seniors who responded “no” to all other respondents who said “no.” Although 
not statistically significant based on Fisher’s Exact test, the result may provide some preliminary 
evidence concerning seniors’ preferences for the flipped classroom (p=0.12). In a related question, 
when asked to compare the use of class time for problem solving or active learning with the instruc¬ 
tor present versus listening to a lecture, 57% of all respondents preferred the former. For the seniors, 
this percentage was only 31%. A test of proportions (based on Fisher’s) for seniors vs. non-seniors 
who preferred in-class active learning was associated with p=0.04. In comparison, Zappe et al. found 
a value in between these percentages, with 48% agreeing or strongly agreeing that they preferred 
problem solving versus lecture during class time [7], 

Content Analysis of Benefits and Drawbacks 

In an open-ended question on the evaluation survey, we asked the students what they liked about 
the flipped classroom and its benefits. The frequencies associated with the categories in our cod¬ 
ing framework are shown in Table 12. The most frequently mentioned benefit as perceived by 44% 
of student respondents related to conveniences afforded by video or online learning, including the 
ability to re-watch videos, self-pacing, flexibility, and accommodation of one’s own preferences. This 
was followed by enhanced or deeper learning, as mentioned by 31% of respondents; this category 
included better understanding and learning, enhanced effectiveness or depth, multiple resources for 
understanding material, and reinforcement and review of material. There were 10% of respondents 
who identified benefits such as higher engagement, better class preparation, and the promotion 
of professional behaviors. This was also a welcome finding. These results were based on a content 
analysis of 389 student responses by a single coder, who was an upper-division undergraduate 
engineering student. A second coder, who was the assessment analyst for the project, coded 30% 
of the responses, corresponding to 118 responses, to provide a measure of inter-rater reliability. The 
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Frequency 

% of 

Respondents 

Category 

Description 

171 

44% 

Video/Online 

Learning 

Re-watch videos 

Work at one’s own pace; pause video 

Flexibility, convenience, own preferences 

Modularization of topics 

121 

31% 

Enhanced 

Learning or 
Learning Process 

Better understanding; less confusion 

Enhanced leaming/effectiveness/depth/ability 

Subject matter retention 

Multiple sources/resources for understanding 

Reinforcement and review 

Multiple attempts 

98 

25% 

Alternative Use 
of Class Time 

In-class active learning, problem solving, clickers 

In-class support and questions 

In-class group time for projects 

Student interactivity and peer support 

59 

15% 

Specific to Course 
or Videos 

Videos concise or had a good pace 

Overall work time less 

Videos had relevant content (e.g., demo or examples) or were of high 
quality 

37 

10% 

Preparation, 
Engagement 
& Professional 
Behaviors 

Engaged during class; paid attention; not bored 

Enjoyed class 

Arrived to class prepared 

Ability to learn on one’s own; independence 

Drove motivation and accountability 

30 

8% 

No Benefit or 
Neutral Result 

No benefits perceived 

Did not like flipped instruction 

Videos not used 

Instructional differences not noticed 


Table 72. Summary of Open-Ended Responses to Benefits. 


inter-rater reliability score based on Cohen’s Kappa was k = 0.75, which suggests good agreement 
beyond chance [51]. 

In a second open-ended question, we asked the students what drawbacks they perceived with 
the flipped classroom and their suggestions for improvement. The same undergraduate engineer¬ 
ing student performed a content analysis of 356 student responses. As shown in Table 13, the most 
frequently mentioned drawback or suggestion involved feedback specific to the particular instructor 
or his videos, such as “include more examples in the video” or “videos did not have an appropriate 
pace.” This was followed by suggestions regarding the use of in-class time with the flipped class¬ 
room (27%) and requests to better prepare, equip, and incentivize students for this new method of 
instruction (17%). Twelve percent of respondents noted increased burden, load, and stress with the 
flipped classroom. We were encouraged to learn that very few respondents (2%) indicated decreased 
learning, and only 9% recommended or wanted a different teaching approach. As with the analysis 
of the benefits, a second coder, the assessment analyst for the project, coded 32% (n=114) of the 
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Frequency 

% of 

Respondents 

Category 

Description 

136 

38% 

Specific to Course or 
Videos 

Include more examples or problems in the videos 

Videos needed editing or bug/technical fixes 

Videos were too long 

Videos were not sufficiently described 

Videos were dry or boring 

Videos did not have an appropriate pace 

Videos repeated information 

Video material was too complex 

97 

27% 

In-Class Time 

Increase time for active learning or problem solving 

Increase effectiveness or relevancy of problems; grade them 
Provide appropriate amount of lecture or content review 

Have more instructor-types during class to assist 

Synchronize class activity and video content 

59 

17% 

Prepare, Equip & 
Incentivize Students to 

Flip 

Prepare students for the flipped learning method 

Incentivize students, including video quizzes 
Clarify/emphasize expectations, including video watching 
Provide video “lecture” notes 

Ensure videos available in advance for students 

55 

15% 

No Drawbacks or Neutral 
Result 

No drawbacks or suggestions 

42 

12% 

Load, Burden, Stressors 

Insufficient time to complete out-of-class activities 

Increased work load 

Increased time burden 

Concerns over grades or impacts to the grade 

Accountability quizzes (including surprise) 

31 

9% 

Approach Differently 

Do not flip courses in general; use traditional teaching 

Do not flip this course in particular 

Provide students with a choice on flipping 

Flip only a portion of the class periods 

18 

5% 

Video/Online Learning 

Students unable to ask questions during a video 

Instructor unable to sense student understanding in a video 
Distractors to viewing videos in a non-classroom setting 

Less motivation to attend class 

8 

2% 

Student Learning 

Lesser understanding or learning 

Difficulty learning from a video 


Table 13. Summary of Open-Ended Responses to Suggestions/Drawbacks. 


responses to provide a measure of inter-rater reliability. The inter-rater reliability score based on 
Cohen’s Kappa was k = 0.83, which suggests strong agreement beyond chance [51], 


DISCUSSION 


Our direct assessment results involving exam and homework scores for flipped versus non-flipped 
instruction across our various courses were mixed. This coincides with other studies in the literature, 
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as discussed the literature review section. In addition, our indirect assessment results indicate, as do 
others from the literature, that some students prefer the flipped classroom while others do not. For 
example, 54% of the North Dakota mechanical engineering students preferred the flipped format 
[23], This was higher than our aggregate result in statics and introductory mechanical design, with 
32% preferring the flipped format. Similar to others who have flipped courses, our mechanical design 
instructor noted in his post-course interview that some students were dis-satisfied with the lack of 
“live” lectures [23], Based on this, he gave a small amount of “live” lecture on SolidWorks during 
the subsequent semester, and he planned to do more, as the students were appreciative. Likewise, 
based on his post-course interview, our senior instructor planned to deliver upfront mini-lectures 
in future implementations of his course, in part because many students came to class not knowing 
how to engage with the problems. Others have considered this same approach [30, 66]. There have 
also been mixed reports in the literature of freshman preferences for the flipped classroom. Only 
22% of our freshmen preferred the flipped classroom. The remainder was equally split between “do 
not prefer” and “unsure of my preference.” With flipped first-year engineering courses, a low level of 
classroom preparation is another theme that has been discussed in the literature [41, 67], Similarly, 
our freshmen did not use the videos for first-time instruction as intended. Our instructors hoped to 
instill in the freshmen a tendency to arrive to class prepared to use their skills and ask questions - 
versus arriving with the expectation of being “given” information. 

We feel that our freshmen did not take responsibility for their class preparation and instead ex¬ 
pected to “be taught” during class. We believe freshmen may not have the maturity to carry out the 
flipped style of instruction as fully as intended, including an inability to manage time well. However, 
in the viewpoint of our freshmen instructor and program director, this does not preclude the desir¬ 
ability of introducing freshmen to the flipped method. Given these maturity issues, though, it may 
not be desirable to flip the entire freshman curriculum. One of his hopes was to promote behavioral 
changes in freshmen, including teaching them “how to learn” and research problems initially on their 
own. He felt that students must be exposed to flipped instruction often during their undergraduate 
careers for it to have a deep impact; thus, flipping should be a school wide initiative. This sentiment 
coincides with the viewpoint of our senior-course instructor, as will be discussed. Thus, preparing 
freshman as well as all students for the expectations of the flipped classroom is a necessary first step. 

One of the predominant themes noted by the instructor of our senior-level chemical engineering 
course was student resistance to and dissatisfaction with flipped instruction, based in large part 
on his teaching evaluations. Unfortunately, these students were engaging with flipped instruc¬ 
tion for the first time as seniors. If they had encountered it previously, they may have been more 
prepared for and satisfied with this mode of instruction. It is also possible that this was related 
to perceptions of a lack of course organization, since the course was being offered for the first 
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time in flipped mode. Similarly, in a senior-level flipped control systems course, there was some 
initial student resistance to and frustration with inverted instruction. However, the students began 
to adjust after about four weeks. One student explained, “It took over half the quarter to finally 
get used to it. I recommend starting this learning style in a different class [30].” Another survey 
comment indicated that the new learning style ‘‘Should be implemented before senior year [30].” 
Although reasons were not provided, according to these authors, researchers are divided on using 
flipped formats in upper division courses [29]. Others have had similar experiences with seniors 
[31], Interestingly, in his post-course interview, the instructor of our senior course felt that flipped 
instruction should not be used with upperclassmen not previously exposed to the pedagogy, 
in line with the goals of our freshman instructor. In the interview, the senior instructor reflected 
that his students had not taken full advantage of the structure of the flipped classroom, as only 
a minority used all of the available face-to-face class time to seek assistance or work on group 
projects. However, he did note an increase in computational ability based on successful models 
developed, compared to prior semesters. He also noted fewer “straightforward” student questions 
and felt that flipped instruction promoted deeper engagement and better career preparedness 
for his most highly motivated students, in line with his objective of increasing deep learning and 
higher order thinking. Thus, despite their resistance, some of our seniors appeared to experience 
academic gains in the course. 

Since we noticed a lack of engagement with and resistance to the flipped classroom by our fresh¬ 
men and seniors, we compared their classroom environment responses to those of the sophomores 
and juniors. The freshmen (n=250) rated every dimension lower in comparison to the sophomores 
and juniors (n=469). Five of the seven dimensions were significantly lower on the order of p<0.0005; 
four were associated with medium or large Cohen’s d effect sizes. The seniors (n=74) rated five of 
the seven CUCEI dimensions lower than the sophomores and juniors did, with three being signifi¬ 
cantly lower on the order of p<0.0005. The associated three effect sizes were large. The seniors 
rated the satisfaction dimension at 2.75 on the five-point scale. This was lower than the average 
satisfaction rating of 3.49 by the sophomores and juniors (p< 0.0005) and was associated with a 
large effect size of d= 0.85. 

Lessons Learned 

One of the challenges that emerged for several of the instructors was serving as one consultant 
to many students during class, with some groups having to wait for assistance as the instructor 
circulated throughout the classroom. This has also been noted by other instructors. The signal pro¬ 
cessing instructor discussed previously assessed that a single faculty member could adequately 
coach about 30 students during class time [31], Our facilities layout instructor concluded similarly 


28 


FALL 2016 





ADVANCES IN ENGINEERING EDUCATION 

Flipping Engineering Courses: A School Wide Initiative 



during his pilot flipping and was able to recruit a teaching assistant who was knowledgeable in this 
subject area, so that two resources could assist the 70 students. In his second semester, this instruc¬ 
tor was able to recruit a second TA, so that three resources were available, showing a nice evolution 
in meeting one of the larger challenges initially identified with flipping the course. 

The online or in-class quizzes taken by our students after watching the videos emerged as help¬ 
ful both for accountability purposes and for providing feedback on conceptual understanding and 
“muddiest” points. Zappe et al. recommended the use of online quizzes before class as an account¬ 
ability check [7], In his first semester of flipped instruction, our mechanical design instructor noted 
a lack of video preparation with some students and began administering quizzes at the semester 
midpoint. His observation was later corroborated by both the self-reported and web analytics data. 
Similarly, the mechanical engineering instructors at the University of North Dakota determined that 
there must be “gate checks,” or assessments to ensure that students arrive to class prepared [23], 

Our instructors found recording the videos to be time consuming and challenging, noting in some 
cases that “lecturing to an empty room” during the recording was more difficult than anticipated. 
We found that the inability to ask questions during an “online lecture” was a small disadvantage; 
this includes student as well as instructor-initiated questions. Another lesson learned was the re¬ 
alization that not all courses or topics may be well-suited to flipping. In a post-course interview, 
our facilities layout instructor identified specific topics that he felt were better suited than others, 
including one that involved the use of layout software. In general, he felt the better suited topics 
tended to be formulaic or mechanical. In their post-course interviews, three other instructors reit¬ 
erated this conclusion about considering the topic or course before flipping it without specifically 
being asked the question. 


CONCLUSIONS 


Active learning in a flipped classroom enables students to apply and practice concepts and skills 
during class with the instructor present to provide coaching and support. At the same time, it pro¬ 
motes lifelong learning skills by requiring students to obtain pre-class exposure to course content 
through video lectures, thereby arriving to class prepared to work. Active learning leads to increased 
involvement in one’s learning, enhanced understanding and outcomes, and a deeper learning ex¬ 
perience, including the higher level skills in Bloom’s taxonomy [8-9, 11], Given these benefits, our 
school of engineering officially promoted the flipped classroom starting in the fall of 2013 across 
its multiple majors. Required freshmen through senior-level courses in introductory programming, 
statics/mechanics, mechanical design, bio-thermodynamics, facilities layout/material handling, and 
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chemical engineering dynamics and modeling comprised an inaugural group of flipped engineering 
courses. To summarize our results, we return to our research questions. 

RQ1: Does the flipped classroom promote student engagement during class, and does it positively 
impact the classroom environment? 

Based on structured classroom observation and comparison against a national study, we were able 
to demonstrate highly active and interactive classrooms within our school-wide flipped classroom 
initiative. The TDOP elements of interest, including problem solving, small group work, individual 
active work, student discussion, and instructor coaching, were significantly higher in our classrooms 
compared to more traditional classrooms. For the statics/mechanics and facility layout courses, we 
were able to perform pre-flip classroom observation. With the statics course, which did not contain 
elements of active learning prior to flipped instruction, our structured observation revealed signifi¬ 
cant increases on the order of p<0.01 in the occurrence of small group work and student discus¬ 
sions. This course displayed an evolution over the academic year in the amount of interactivity and 
active learning during class; thus, instructional changes such as course flipping may require time 
for full implementation. With facility layout, there were also significant increases (p<0.01) in the 
percentages of problem solving, student discussions, and instructor circulation to answer questions 
when considered pre-flip to flip. Thus, in both courses, there was an opportunity for more student- 
centered, active learning during class, and our structured observation showed this opportunity to 
have been met by the flipped classroom. Based on these various comparisons, we conclude that 
flipped instruction drove student engagement and involvement in our engineering classrooms. 

For two of the courses - statics/mechanics and facilities layout - we were able to administer a 
pre-flip environment survey. In statics, the flipped environment was rated as significantly better 
(p<0.0005) on four CUCEI dimensions compared to the pre-flip environment. These dimensions - 
student cohesiveness, individualization, involvement, and personalization - are key goals of the flipped 
classroom. The effect sizes were also large. In the facility layout course, there were also significant 
improvements in individualization and personalization (p<0.0005) as well as involvement (p=0.002). 
Based on nearly 800 classroom learning environment responses, the flipped classroom in our school 
was rated highest on the personalization dimension, scoring 3.88 on the five-point CUCEI scale. 
This dimension assesses interaction between the students and instructor; thus, students perceive 
student-faculty interaction as a strong aspect of the flipped classroom. Our school-wide CUCEI data 
compared favorably with other CUCEI studies of STEM classrooms in the literature. Thus, we conclude 
that flipped instruction had a positive impact on the environment in our engineering classrooms. Our 
freshmen and seniors, however, did not evaluate their flipped classroom environment as favorably 
as the sophomores and juniors did. 
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RQ2: Is the flipped classroom associated with increased student achievement and learning of 
content? 

Exam, homework, and grade results in flipped vs. non-flipped courses at other universities have 
been mixed, as were ours. Two of our courses had statistically significant improvements in exams 
or homework and two showed statistically equivalent results. However, several of our instructors 
noticed improvements in higher-order thinking skills and content proficiency with flipped instruction. 
Our bio-thermodynamics instructor identified improvements in problem solving, including solution 
of more advanced problems, in his flipped class. Our SolidWorks instructor found that students in 
the flipped course were more sophisticated, proficient SolidWorks users; he attributed this to more 
practice time, which was possible in the flipped classroom. We found a correlation of approximately 
0.40 between the number of videos accessed and the course grade for our sophomores, juniors, and 
seniors, suggesting some relationship between preparation in the flipped classroom and achievement. 
There was a very low correlation for the freshmen, which was not significantly different from zero. 
Based on two separate sources of data, the freshmen in large part did not make use of the videos 
as first-time instructional tools, as intended. Based on these same sources of data, the sophomores 
through seniors took greater responsibility in watching the videos before class. This suggests that 
freshmen may require additional coaching and monitoring in the flipped classroom, especially if 
they have not previously experienced this style of instruction. We found other examples in the lit¬ 
erature in which preparation was an issue of concern in freshmen flipped classrooms. In addition, 
we encountered resistance by our seniors to flipped instruction and likewise uncovered examples 
in the literature of resistance by seniors. 

RQ3: What strengths, benefits, and drawbacks do students perceive with the flipped classroom? 

The flipped classroom evaluation survey showed an approximate three-way split in student prefer¬ 
ences for the flipped classroom, with 27% indicating a preference, 36% indicating a non-preference, 
and 36% being unsure. The students nonetheless saw value in this method of instruction, as 57% of 
respondents preferred using class time for problem solving or active learning with the instructor 
present versus listening to a lecture. In an open-ended response, the most frequently stated benefits 
of the flipped classroom included flexibility and convenience-related benefits (44% of respondents), 
enhanced learning or learning process (31%), and alternative use of class time (25%). There were a 
small percentage of respondents (10%) who identified preparation, engagement, and professional 
behaviors as a benefit. 

We believe that flipped instruction is an effective learning approach, despite some challenges we 
encountered. Like all pedagogies that challenge the status quo, there are issues, aspects, and chal¬ 
lenges that must be considered and understood. Flipped instruction promotes more active learning 
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and responsibility for life-long learning, student engagement, interaction in the classroom, collabora¬ 
tion, individualization and flexibility, preparation, and more real-world engineering experiences in the 
classroom. Over the past academic year, the members of our faculty learning community have consid¬ 
ered, and in some cases re-thought, their approaches with this pedagogy and have reflected on other 
lessons learned. In the process, we discovered the advantage to introducing the flipped method of 
instruction early in the undergraduate career. Although our freshmen did not engage with the flipped 
classroom as intended in terms of the video usage, introducing this method of instruction to them as 
freshmen likely better prepared them to engage with it in their sophomore years and beyond. Thus, 
one of the instructors’ main takeaways was the need to educate students upfront about the expecta¬ 
tions and goals of the flipped classroom, which we aptly did by developing a student-produced video 
on flipping. Going forward, this video will be distributed at the start of the term to orient students 
to their new learning environment and prepare them for the expectations of the flipped classroom. 

All but one of the instructors continued to use the flipped instructional style in the course. Our 
senior-level instructor decided not to flip the course the following term but would like to return to 
this teaching style in the future. We continue to promote and formally support the flipped style in 
our school of engineering, and additional instructors continue to implement the approach to drive 
active learning and in-class engagement with their students. 
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